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ABSTRACT 



A model for evaluating an educational program for 
disadvantaged families (the Expanded Food and Nutrition Education 
Program, EFNEP) is presented and discussed. The steps in program 
evaluation are given; these steps ars: Rationale; Goals, Intents, 
Objectives; Antecedents; Transactions; and Outcome. The collection of 
data is described, and the use of content analysis in EFNEP is 
discussed. The three major elements of evaluation are given as 
criteria, standards, and judgments. The three types of criteria 
necessary for evaluating evaluation information are said to be 
scientific, practical and prudential. It is concluded that in order 
to improve the level of evaluation being done in adult education, 
adult educators need to evaluate their own evaluation efforts and use 
the results to imorove future efforts. References are Drovided. 
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While most adult educators agree on the need for program evaluation , 
in practice, there Is little consensus as to the essential characteristics 
of such evaluation. Only in recent years has the adult education profes- 
sion begun to develop it f s own literature on evaluation* Previously we 
just adapted the literature (and very often the philogophy and methodology) 
of public school evaluation to adult education programs. Early evaluation 
efforts in adult education focused on attendance at meetings or adoption 
of practices* Evaluation in terms of program objectives has also received 
emphasis and recently evaluation aimed at program improvement has received 
some attention. 

As adult educators become more involved in social action type programs, 
the need for a different approach to evaluation becomes apparent* Nest 
social action programs are not single purpose or homogeneous; therefore, it 
is necessary to evaluate several aspects of the program. Adequate measurement 
of a social science, concept frequently requires the use of more than one 
indicator since the concepts are often multidimensional. A comprehensive 
approach to evaluation must be taken if substantial program improvement 
is to result (Knox, 1969). 

In discussing some of the concerns in evaluating social action programs, 
Berlak (19 70) s tated : 

"The central purpose of evaluating most social action programs is the 
broad measurement of change. Evaluation is a comparative and historical 
enterprise which can best be carried out as a part of a general effort to 
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measure educational status and change. Alms of social action programs are 
diverse, and their purpose Is to shift the position of specified target pop- 
ulations relative to the rest of society; their evaluation cannot be ac- 
complished by Isolated studies of particular aims with Inappropriate standards 
of comparison. Evaluating broad social action programs requires comparably 
broad systems of social measurement . ” 

The complexity of social action programs points up the need for broad 
systems of social measurement for measuring change. Since many of these 
social action programs are experimental in nature, the adult educator is 
Interested not only in measuring change, but also in determining what is 
happening in the program and which methods are working so that this information 
can be used for program improvement. 

DEFINITION 

The definition of evaluation in education has gone through many cycles. 

An early definition equated evaluation with measurement (Thorndike and 
Hagen, 1961.) Another widely accepted definition of evaluation has been 
izhat of determining the congruence between performance and objectives (Tyler, 
1950). Another way of defining the concept is to say that evaluation is 
professional judgment. This is exemplified by the visiting team of experts 
who come to view a program and make a judgment. The judgment is the evalua ti on. 

Evaluation is a mechanism with which the character of an educational 
enterprise can be explored and expressed. A definition for evaluation 
which can serve as a framework for a model to evaluate social action programs 
and educational programs for the disadvantaged needs to be coop rehens ive . 

Stuf flebeam (1971) defines educational evaluation as the process of delin- 
eating, obtaining and providing useful information for judgi n g decision 
alternatives. 

Stake (1967*) presents this view: 
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"Curriculum evaluation requires collection, processing, and Interpre- 
tation of data pertaining to an educational program. For a complete 
evaluation, two main kinds of data are collected: (1) objective descriptions 
of goals, environments, personnel, methods and content, and outcomes ; and 
(2) personal judgments as to the quality and appropriateness of those goals, 
environments, etc." 

In his definition Knox (1969) focuses on program Improvement: 

"Continuous program evaluation Is the process by which evidence regarding 
program effectiveness Is systematically collected, analyzed, and used to 
improve programs of continuing adult education." 

Each of these definitions points to the need for evidence to be collected 
which can serve as a basis for decisions regarding educational programs. (Cohen. 
1970) The decisions which need to be made cover a wide range including fund- 
ing decisions as well as decisions regarding changes which need to be made 
to improve programs. 

Scrlven (1967) discusses two roles for evaluation. The first, formative. 

Is designed to discover deficiencies and successes In the curriculum or program 
as It Is in operation. Data are used to make judgments about what works when 
an educator Is trying to make his ideas or ideals come about. It Is often 
designed to answer why one thing works and some thing else doesn't. Formative 
evaluation Is useful for program Improvement since findings can be used as a 
basis for changing, maintaining or inproving the program. The second role . 
which he svomatlve. Is designed to make an assessment of the f in a l 

product. It is the evaluation which Is done after a program Is completed to 
determine what happened. 

Dxt authorities do not use the terms evaluation and research synonymously. 
So chman (1967) distinguishes between evalua ti on and evaluative research. Be 
refers to evaluation «• the social process of making judgments of worth. Be 
says evaluative research utilizes scientific research methods sad techniques 
to make an evaluation. Evaluative research refers to those procedures for 
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collecting and analysing data which increase the possibility of "proving" 
rather than "asserting.” 

I am assuming that evaluation should be logical * empirical and objective. 
There may be occasions when evaluation can also be scientific* at other 
times this may not be possible and the nature of the evaluation may not demand 
It. Very often an evaluator is examining a specific program in a specific 
community and the degree of generaliz ability is limited. He may be describing 
what is rather than probing for why. Evaluation is not of lesser Importance 
than evaluative research. Each has their place. In the rest of this dis- 
cussion* as I use the term evaluation I realize that there are various 
levels of evaluation. 

MODEL 

In this section a model will be presented for evaluating an educational 
program for disadvantaged families. The Expanded Food and Nutrition Education 
Program (EFHEP) is a federally funded program conducted by the Cooperative 
Extension Service. The program is in its third year of operation in Wisconsin. 
The objective of EFNEP is to improve the dietary level of low income families 
through education and Improved use of resources. The achievement of Oils 
objective should give the participants a greater degree of control over their 
own lives. 

The important innovation in this program has been the use of the para- 
professional. It was recognized that the paraprofesslonal could communicate 
with the program audience more readily than the professional, ha most 
instances the paraprofesslonal begins her work with the families on a one- 
to-one basis. 
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This brief description of the EFNEP program points up the variety of 
dimensions which are a part of the program. An evaluation model is needed 
which will allow the e valuator • to focus on several dimensions which will 



provide information to be used in making decisions regarding the program. 



Such a model would seem to have several purposes: 

1. It forces the evaluators to formulate a broad conceptualization of 
the program. 

2. Reminds the evaluator of the many facets which must be considered 
in relation to the program. 

3. It offers some direction to the evaluation process. 



R 
A 
T 
I j 
0 
U 
A 
L 
E 



4. It suggests the kinds of data which need to be collected. 

5. Provide a framework for reporting findings from evaluation study. 

* Model for Evaluating EFNEP Program 

Goals Collection Criteria * Program Isprovnnent 

■^Intents — ■ ■ ■) of jjv Standards '■“^■Judgnents^' Program Accountability 
Objectives Data 

Antecedents 

Transactions 

Outcomes 



DISCUSSION 

Rationale 

Any program evaluation needs to begin with a statement of the p r o g r a m 
rationale. The rationale catca the philosophic background and basic 
purposes of the program. The EFNEP is an outgrowth of national concern about 
poverty and hunger. Nutrition education is the vehicle being used to reduce 
the incidence of poverty end hunger. It is recognized diet other problems 
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may have Co be handled before nutrition education can begin and nutrition 
education may point to other problems needing attention. Through the employ- 
ment of the Indigenous paraprofesslonal It Is hoped that families from the 
target audience can be reached more readily and chat the c omm unications will 
be easier. The Income which the paraprofesslonal receives should give her 
additional economic power which theoretically should give her greater control 
over her own life. The evaluator needs to be cognizant of the program rational# 
as he formulates his evaluation plan. The rationale should provide one basis 
for Identifying program goals, objectives or Intents. 

Goals, Intents, Objectives 

The goals. Intents or objectives of the program are drawn f rom the 
program rationale. The stated Intents or goals as well as the unseated ones 
should be Included In the evaluation. The of Che learners as well as 

these of the educators should be considered. It must be recognized that 
those won't always be congruent. Examples of the goals Included In EFNEP 
evaluation are: 

- ability to manage resources 

- Improved nutritional knowledge 

— Improved food consumption patterns 

— Improved family relations 

- Improved family health 

- ability to deal with problems not of a nutritional nature 

- gaining control of life 

- establishing a helping relationship 

- Increasing problem solving skill 

- Increasing individual growth and development 

- ability to be better consumers 

Ail of these goals would be examine d In relation to the families participating 
In the program and many of would also be examined In relation Co the 

employed paraprofesslouals. 

Ante ceden ts 

Antecedents are t h ose conditions existing prior to learning and tend to 
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ba thought of as static* Many authorities refer to them as entry behavior 
levels or as a description of ’‘what is” when the program begins* In EFNEP 
this description needs to include data on the learners, the para-professionals , 
the professionals and the programming situation in the county* Examples of 

antecedents to be described are: 

- econ o mic characteristics of the learners 

- social and physical environment of the learners 

- food consumption pattens of learners and paraprofessionals 

- economic characteristics of paraprofessionals 

- indigenous or tfb lqul tous -paraprof esoionals 

- level of nutrition knowledge of paraprofessionals 

- experience cf professionals in this type of program 

- staffing patterns in the county- 

Transactions 



Transactions are dynamic and include those activities in which learners 
and teachers engage, together with the instructional materials used* The 
physical and psychological environment in vMcb learning occurs is important. 
A vide variety of transactions need to be examined in relation to EFNEP: 

- relationship between paraprofessionals and families* 

- relationships between paraprofessionals and professionals* 

- relationship cf paraprofessionals, professionals and volunteers. 

- how paraprofessionals use teaching materials. 

- lea rn i n g situations provided for paraprofessionals. 

- learning situations provided for professionals. 

- ways in which paraprofessionals work with families -in dividuals-gro up . 

- involvement of volunteers in the program. 

- channels for decision making within the program. 

- when it is appropriate to move families to another phase of the program. 

- communi cat! on links between: 

paraprofessionals and families 
paraprofessionals and professionals 

parapr ofesslonals , professionals and other community agenc i es 
professionals and commulty influentials 
professionals and co-workers within the organisation 




Outcomes 

Attention will be given to both intended and observed outcomes. The 
Intended outcomes relate back to the goals of the program. If different out- 
comes are observed as data are collected, they will be recorded and considered. 



7 



8 



O 

ERLC 



Observed outcomes take on additional importance when the concern Is with 
evaluation for program Improvement. 

COLLECTION OF DATA 

Some of the data needed for evaluation of EFNEP la available from 
reports and Internal data systems . Methods need to be devised to collect 
data on attitudes, relationships, levels of knowledge ‘ practices followed, 
conmunlcatlons links and the other data, which the goals, antecedents and 
transactions Indicate are needed. Interview schedules, questionnaires, case 
studies and content analysis are all methods which can be used to collect 
these data. 

For those concerned about collecting data on changes In attitudes , 
knowledge or behavior, evaluation of social action type programs present 
some methodological problem . (Cain and Hollister, 1969, Suchman, 1967) It 
Is particularly difficult to predict long run or permanent behavior changes 
from the short term Indicators revealed by an on-going or just completed 
program. Outcomes which we want to measure may occur months or even years 
after participants have completed the program. Multidimensional measures 
are often needed to evaluate social action programs. 

Ihose concerned about doing higher level evaluation must decide If the 
use of experimental and control groups are a valid way to measure change. 
(Suchman , 1967, Cain and Hollister, 1969) Many educators raise e thica l 
questions about Ala method of evaluating social action programs. For example, 
is It ethical to withhold a program from a group who needs it (control group) 

In order to be able to evaluate the results of a program. Su chman (1967) 
examined this Issue and presented an alternative, the longitu din a l study, 
which has relevance for social action evaluation . 
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The longitudinal design permits comparison of effectiveness of a program 
over a period of time. By making evaluations at different points In time 
one may check on die progress of the program toward Its objectives and, at 
the same time, use the earlier measurements of subjects as a form of self- 
control against subsequent measures. 

A variation of this design would involve the use of different populations 
at different stages of the program. If these successive cohort groups were 
representative of the same target population. It would be possible to evaluate 
effectiveness at different points In. time. This design Is also a more efficient 
design when It becomes difficult or Impossible to keep In contact with the 
original subjects , ns may be the case in many social action programs . It Is 
my expectation that this variation of the longitudinal study will be used 
as we try to measure whether behavior change has occurred through EFNEP. 

Content Analysis 

Content analysis Is a research technique for die objective, systematic 
and quantitative description of the manifest content of coamunlcatlons . 
(Berelson, 1951) Stake (1970) says the thorough evaluator attempts to analyze 
the documents of the comaunlty, the newspaper and the minutes of meetings 
to learn how Ideas and values have fared across time. Content analysis Is 
die technique which can be used to exanriLne these documents. Dickinson and 
Rusnell (1971) used content analysis to examine die content of ADULT EDUCATION 
JOURNAL to ascertain trends and patterns In die content of the journals as 
Indications of the development of the discipline of adult education. 

Content analysis must be objective, systematic and quantitive. Objectivity 

* 

stipulates that the categories of analysis should be d efine d so precisely that 
different analysts can apply them to the same body of content and secure the 
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same results. Content analysis needs to be systematic as It Is designed for 
the establishment of scientific propositions. It Is quantitative as It notes 
the extent to which the previously defined categories appear In the content. 

We have used content analysis in EFNEP on the written logs which each 
paraprofesslonal writes after each visit to a family. A sample of logs cover- 
ing at least ten visits by the professional to one family was used for analysis 
purposes. Four categories were generated: 

1) Behavior toward: paraprofesslonal, food, food preparation, family , 
self, money; 2) Working relationship; 3) Skills In: meal preparation, 
meal planning, shopping; and 4 } Problem solving: food preparation, 
equipment. 

Three judges were trained to do content analysis of the logs. Trai nin g 
was necessary to establish reliability in their scoring. Each judge reads 
the logs, indicating the frequency of each category In the log, and assigning 
a positive, negative or neutral value to the category each time it is noted. 
Quantitative measurement is also noted on references to nutrition content. 



communications methods used and other problems mentioned. The mean of the 

judges* scores will be used to give a final score to each log. 

These examples from logs will illustrate the use of content analysis: 

T, Mrs. X had been working at hospital as an aid but she left. All she 
was earning went for food as she had less time to bake and cook and was 
buying more prepared foods. ^ V7e made cupcakes. She has 3 burner gas 
stove and small oven. Only had 1 cupcake pan so we used broiler pan 
and cxip cake papers. 2 Mrs. X did not measure ingredients when she baked. 
The cupcakes turned out good.** 1 



1. Behavior toward food preparation (neutral) 

2. Problem solving - equipment (+) 

3. Skill in food preparation (neutral) 

4. Skill in food preparation (+) 



f, IIrs. P. lost right hand at work. She thinks people don*t want to see 
her. She don’t know how to cook much. ^ She had everything out for us 
to make baked custard. 3 While custard was baking I asked her what 
nutrition this dish have. She answer calcium from the milk but she 
didn’t know about the eggs.** 
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1. Behavior toward self (-) 

2. Skill In food preparation (-) 

3. Working relationship 

These are Isolated examples from the logs* As a series of logs relating 
to these families Is analyzed you would hope to see trends developing* 

Content analysis of logs is a time consuming process but it has several 
advantages* It gives an indication of growth (behavior change) in the four 
categories through the progression of visits. It also provides an indication 
of gaps in content, attitude or methods which can be corrected through train- 
ing of the paraprofesslonals. 

Interview Schedules 

We are currently using interview schedules with the paraprofesslonals 
and with program families* The interviews are being conducted by professional 
survey research interviewers* The schedules were designed to collect data 
relating to the goals and transactions mentioned earlier* The interview 
schedules were pre-tested before being used in the field* 

By comparing data collected through the Interviews with base data col- 
lected when families began the program it will be possible to determine if 
behavior changes have occurred in food consumption patterns* Much of the 
data on transactions can be used to make changes in that portion of the program 
which will lead to program impr o vement * 

One of the concerns of professionals involved with EFNEF is that too 
many families stay in the program longer than necessary after they have learned 
basic knowledge and skills* From the data we get from interviews with program 
families we hope to be able to establish some criteria which can be used to 
identify steps of progression for families in the program* 
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Crittrli, Standard* «nd Judgments 

Criteria* evidence and Judgm en t are three major elemanta of evaluation. 
Without criteria or standards * Judgment cannot take place. Steele «d Maes 
(1970) have discussed the criteria problem In adult education program evaluation 
The criteria which would describe the BF8BP goals discussed earlier cmi be 
drawn from several sources. Research literature, authorities and personal 
experience are all valid criteria sources. 

It Is recopilzed that personal and institutional values will influence 
the definition of criteria. In EFNEP evaluation we have involved the pro- 
fessional, and In some cases the paraprofessionals » In helping to define the 
criteria which will be used in raking pro gra m judgments. This process of 
defining criteria can be used to further clarify goals end intents and to 
examine the relevance of values to the situation. In foxmativa evaluation 
as much of EFHEP evaluation Is, we feel the evaluation experlenca Itself 
can be a positive learning experience for the professionals and p ara profea- 
slonala Involved end can result in program Improvements. 

After criteria ere defined mid evidence collected. Judgments need to be 
made. In some instances the evaluator nay make the Judgments sad pressnt 
then to the d e ci s ion makers. In other Instances It may be appropriate to 
Involve the program decision makers la making Judgments. In our evaluation 
we hope to involve the program decision make re in making the Judgments. 

EVALDATIHC THE EVALUATIOK 

Any evaluation effort must conform to certain criteria to justify being 
celled acceptable. Stufflabeam (1971) discusses three types of criteria for 
evaluating evaluation information; scientific, practical and prudential. 
Scientific criteria Includes internal end external validity, r e l i a bi l i ty end 
objectivity. 
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Evaluation information nae bo informative eo cho roeolvor. Iho practical 
crltorla Stufflebeem (1971) dlacuMoa la ehla roapoct oro: relevance of data 

to cba purpoeas of tba evaluation ; laporcanca of tbo data to the ova lu at l oo 
purpooaa; doaa tba information hava aufflclaat acopo to bo useful: la tba 
laforaotloa credible; la the lafonatloo timely; and la tba Infonatloe being 
disseminated to all who need to knew. 

Tba prudential criterion Stuf f^ebaaa (1971) llata la efficiency. TIm, 
coot and personnel needed are all efficiency factor* which need to be considered 

Those adult educators Involved In pro g r am evaluation usually spend their 
time planning end conducting such evalnsrlone end do nor give adequate atten- 
tion to bow to l a y rov e evaluation efforts. If we are to lop rove our evaluation 
efforts In adult education we ne ed to ettond to the criteria dtac uaaed above. 
Finally 

Program Improvem ent and p rogr a m accountability are both legitimate out- 
comes for evaluation of adult education p r o g r am s. As adult ed u ca t ors become 
■ore involved in model action type programs, e more coayro ho aolva approach to 
evaluation must be taken, la selecting or fo rm ul atin g a model for evaluating 
these kinds of p r o gres s adult educators need to be specific about what they 
are doing but tbey also n ee d to be elect to things wh o s e relevance la not 
clearly known ea yet. Plana need to be specific enough to show diet they 
went moat to describe, discover end communicate but open enough to report 
the unexpected. In order to i m prove the level of evaluation being dooe in 
adult education, adult educators need to evaluate their own evaluation efforts 
sod use tbs results to l e p rou s f utu re efforts. 
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